CANONICAL SYSTEMS AND THEIR LIMITS ON STABLE CURVES 



ZrVRAN 



Abstract. We propose an object called 'sepcanonical system' on a stable curve Xq which is to serve as limit- 
ing object- distinct from other such limits introduced previously- for the canonical system, as a smooth curve 
degenerates to Xg. First for curves which cannot be separated by 2 or fewer nodes, the so-called '2-inseparable' 
curves, the sepcanonical system is just the sections of the dualizing sheaf, which is not very ample iff Xg is a 
Umit of smooth hyperelliptic curves (such Xq are called 2-inseparable hyperelliptics). For general, 2-separable 
curves Xq this assertion is false, leading us to introduce the sepcanonical system, which is a collection of linear 
systems on the '2-inseparable parts' of Xg, each associated to a different twisted limit of the canonical system, 
where the entire collection varies smoothly with Xq. To define sepcanonical system, we must endow the curve 
with extra structure called an 'azimuthal structure'. We show (Theorem |7.5) that the sepcanonical system is 'es- 
sentially very ample' unless the curve is a tree-like arrangement of 2-inseparable hyperelliptics. In a subsequent 
paper 1 13 [ we will show that the latter property is equivalent to the curve being a limit of smooth hyperelUptics, 
and will essentially give defining equation for the closure of the locus of smooth hyperelliptic curves in the 
moduli space of stable curves. 
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Introduction 

A big part of the geometry of a nonsingular curve revolves around its canonical system and canonical 
map, which is always an embedding except for the well-understood exception of hyperelliptic curves. 
The purpose of this paper is to identify and study an appropriate 'limiting object' of the canonical system 
as the curve degenerates to a stable one, an issue that is closely related to the extrinsic geometry, especially 
defining equations, of the closure of the hyperelliptic locus in the moduli space of stable curves. 

The default choice for the limiting object is certainly the canonical system itself, i.e. the linear system 
associated to the dualizing sheaf, on the limiting stable curve. This choice will be analyzed in §1-3 below. 
What the analysis shows is that the canonical system on a stable curve Xq is a good choice of limit as long 
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as Xq is '2-inseparable' in the sense that it cannot be disconnected by removing 2 or fewer nodes. We will 
show (Theorem |3.12| | that a 2-inseparable curve is a limit of smooth hyperelliptic curves iff its dualizing 
sheaf is not very ample. Those curves were classified long ago by Catanese [6|. 

Going beyond 2-inseparable stable curves Xq, it becomes less than clear that the canonical system on Xg 
is the correct limiting object: for example it is no longer true that Xg is the limit of hyperelliptics whenever 
its canonical map is not birational; indeed the locus of stable curves of genus g whose canonical map is 
not birational has components of fixed codimension c independent of g contained in the boundary (in 
fact, in the boundary component Aq) of the Deligne-Mumford Moduli space Mg- One the other hand, 
the analysis of 2-inseparable curves suggests viewing these as atoms for the purpose of constructing the 
limiting object for a general curve. 

Before proceeding to describe our limiting object, it should be mentioned here that there exist gen- 
eral constructions for limit objects of linear series, such as the one published in an Inventiones paper by 
Eisenbud and Harris (cf. [8J) and an unpublished one [14]. The Eisenbud-Harris construction is studied 
in much detail for the case of the canonical series, especially on curves comprised of two components 
intersecting in generically positioned points, in an Inventiones paper by Esteves-Medeiros |9| (which also 
references IMJ). But this limit linear series is not a good limiting object from our perspective, because it 
does not vary smoothly, i.e. it jumps, in families (even locally trivial ones), and therefore does not appear 
to be useful for the sort of enumerative applications that we have in mind (see below). 

The limiting object that we propose here is called the sepcanonical system. It is essentially a part of the full 
limit series associated to the canonical series (for a suitable modification of the family): more specifically, 
the largest part that never jumps. To define the sepcanonical system the curve first has to he endowed 
with some additional structure, closely related to Maine's notion of enrichment [12J, which we call an 
azimuthal structure, and which consists essentially of a collection of smoothing directions at separating 
pairs of nodes (so-called 'biseps') . The composite object is called an azimuthal curve, and the parameter 
space for these is a certain blowup of the moduli space, in which the locus of 2-separable curves becomes 
a divisor. This notion is best behaved for curves of 'semi-compact type', meaning that distinct separating 
pairs of nonseparating nodes (called 'proper biseps') are disjoint, because then the smoothing directions 
may be chosen independently Things are more complicated for non-semi compact-type curves, but this 
turns out not to be much of a problem because we shall see that in a number of specific ways, those curves 
behave non- hyperelliptically. 

Given an azimuthal curve Xq, the sepcanonical system \ojxq\^^'^ is defined as a collection of linear 
systems | coxq \ y ^ containing | coxq \ r, one on each '2-component' Y of Xg, i.e. subcurve obtained by blowing 
up the separating nodes (called 'seps') and biseps of Xg. The apparently arbitrary definition is justified, in 
part, by Theorem l7.5l which shows that in a versal deformation of Xg, each \ coxq occurs as the restriction 
on y of the limit of a certain twist (by a linear combination of boundary components depending on Y) of 
the relative canonical bundle. Still one may ask 

• why the particular choice of twist ? 

• in what sense do the individual |a;|Y^ for different Y form a whole? 

As to the first question, the choice of twist is dictated by by the requirement, motivated as mentioned 
above by enumerative applications, that lo^Xoly^ should be as large as possible (in particular it should 
contain |a;X(,|y), but should never jump as Xg deforms keeping Y intact: in particular the dimension of 
IwxqIy^ depends only on the combinatorics. The fact that enlarging |a;xolY^ must come at the expense 
of Y's neighbors severely limits the possible twist, leading to our particular choice (which, as mentioned 
above, differs from choices made previously, e.g. in |8| and HU). 

The second question is answered by a result in |13| which shows that the various lo^Xoly^ together 
effectively arise from a single bundle map, the so-called azimuthal modification of the Brill-Noether map. 
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Theorem |7.5| in turn is a consequence of an elementary 'Residue Lemma' IS.ll which for certain twists of 
the canonical bundle identifies the sections on the special fibre which lift to the general fibre. 

The main justification for the sepcanonical system is provided by a result in [13 1, based on this paper, 
which extends Theorem 13. 121 to the general case and characterizes limits of smooth hyperelliptic curves. 
As a step towards this result, we prove here a characterization (Theorem l7.5b of stable curves whose sep- 
canonical system is not 'essentially' very ample (e.g. non-birational) on some 2-component, extending 
Catanese's result for the canonical system: namely, those curves are trees of hyperelliptic 2-mseparables 
joined at Weierstrass points and hyperelliptic divisors. These are precisely the curves which are an ad- 
missible double cover of a rational tree. 

An alternative (enumeratively-qualified) approach to limits of hyperelliptic curves (and more general 
1-dimensional linear systems) focuses on the hyperelliptic pencil and its limits (as maps to rational trees), 
rather than the canonical system. This is based on the theory of admissible covers due to Harris-Mumford 
lUlJ and its generalizations such as the notion of relative maps developed by Faber and Pandharipande 
IITOl . As mentioned above, another approach to the limit of the canonical series is due to Esteves-Medeiros 
[95, based on [8]. Some of the lemmas and propositions in §2-4 are similar in substance, if perhaps not in 
point of view, to results published previously. Also, our notion of curve with azimuthal structure is related 
to the notion of 'enriched curve' developed in an impublished Harvard thesis by L. Maino | [T2 | (which in 
turn references our unpublished preprint [14J). In general, the notion of emiched curve, in which biseps 
play no special role, is different from that of (regular) azimuthal curve, in which they do; however, the 
two notions coincide for semicompact-type curves. 

This paper was originally part of (T^, but is being published separately due to length considerations. 

I thank Gwoho Liu for helpful comments, and David Eisenbud for sending a copy of Maine's thesis. 

1. Canonical systems 
In this paper, all curves are nodal unless otherwise stated. 

A semistable curve is a nodal curve X whose dualizing sheaf cvx has nonnegative degree on each 
irreducible component; equivalently, the desingularization of each rational component of X contains at 
least 2 node preimages; a semistable curve is stable if the degree of lox is positive on each irreducible 
component. A stable pair or stable pointed curve is a pair consisting of a semistable curve X and a smooth 
point p on it so that 0Jx{p) has positive degree on each irreducible component. Note if is a separating 
node on a stable curve X, then 

(X,0) = (lX, L0) U (rX, R0) 

with each pair (*X,* 6) stable (as pair). We call the (*X,* 9) the left and right parts of X with respect to 6, 
also denoted rX(0). 

2. Inseparables and base of the Canonical system 

The results of this section are not new (see e.g. [6J) and are included merely for completeness and be- 
cause we have a different viewpoint that will continue later in the paper. We will deal with the base locus 
of the canonical system. This will turn out to be related to separating nodes. We begin by considering 
curves without separating nodes. 

A semistable curve is said to be separable if it has a separating node, inseparable otherwise. The separation 
of a nodal curve X is the blowup X^^P of X at all its separating nodes, called seps for short. Clearly the 
connected components of X^'^P are the maximal connected inseparable subcurves of X, called its inseparable 
components. Any irreducible component is contained in a unique inseparable component. The separation 
tree of a (connected nodal) curve X is the graph, necessarily a tree, whose vertices are the inseparable 
components of X and whose edges are the separating nodes of X. More generally, we may associate a 
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separation tree to any collection of separating nodes on X. Separability is closely related to freeness of the 
canonical system, as the following Lemma begins to show: 

Lemma 2.1. The canonical system of an inseparable connected semistable curve of genus > is free (of base points). 

Proof. We may assume X is singular, and let X' — > X be the blowup of a (nonseparating) node 6, with node 
preimages pi, p2- Then X' is a connected nodal curve of genus g — 1 and h^{cox'{pi + Pi)) = > ^'^('^x')- 
From the definition of dualizing sheaf it follows that cox'{pi + pi) has a section nonvanishing at both 
Pi' Pi hence cox is free at 9. Thus cox is free at all (automatically nonseparating) nodes. It follows easily 
that any base point of lox must occur on a smooth rational components C, each of which must contain 
r > 2 nodes. Consider the restriction map 

p:H°{cvx)^H'{cvx\c)=H°{u>c{r)). 

Its kernel may be identified with the differentials on the complementary subcurve to C, which has genus 
g — r + 1. Therefore p has rank r — 1, hence is surjective, so cox has no base points on C. Therefore lox is 
free. □ 

A smooth rational inseparable component of a nodal curve X is called a a separating line of X (see |3l. 
Definition 4.7) and X is said to be a comb if it has a separating line. The following result is due to Catanese 
( [61/ Thm. D), who uses a different terminology. 

Corollary 2.2 (Catanese). The base locus of the canonical system on a semistable curve is the union of the separat- 
ing lines and the separating nodes. 

Proof. To begin with, it is elementary from Riemann-Roch that the base locus contains the separating lines 
and the separating nodes. For the converse, we use induction on the number of irreducible components 
of the curve X, assumed connected and of genus > 0. If X has a separating line, we can conclude easily 
by applying induction to the remaining inseparable components. Else, let Y be an inseparable component 
that constitutes an end vertex of the separation tree of X. Then X = Y Up Z with Z cormected and 
semistable and without separating lines, and H'^{loy) = H^{coy{p)) again by Riemann-Roch. By Lemma 
12.11 the base locus of cox on Y is exactly p. As H^{cox) ~ H^{coy) ® H^{coz), the proof is completed by 
applying induction to Z. □ 

A pair of smooth points on a nodal curve is said to be separated if they lie on different inseparable 
components. 

Corollary 2.3. A separated pair of smooth points, not on any separating lines, of a connected stable curve, impose 
independent conditions on the canonical system. 

Proof. If p is a smooth point on a non-separating-line inseparable component C of X, there is a by Lemma 
I2.1l a differential on C nonzero at p, and using the natural map H^{u}c) — > H^{cox), this may be extended 
to a dualizing differential on X that is zero on X — C. □ 

An inseparable semistable curve X is said to be PHEL (acronym for pseudo-hjrperelliptic), if the canon- 
ical system \lox \ fails to define en embedding. This includes all semistable curves or arithmetic genus or 
1. A stable pair (X, p) with X inseparable is said to be PHEL if |a;x| is ramified at p. This terminology is 
temporary because we will see later that PHEL is equivalent to another notion, to be defined below even 
for possibly separated curves, that we will call 'hyperelliptic'. 

Lemma 2.4. Suppose (X, p) is PHEL , inseparable, and singular. 

(i) IfX is irreducible of arithmetic genus > 1, the canonical system ofX defines a 2:1 map to a smooth rational 
curve, ramified on p. 

(ii) IfX is reducible and p lies on an irreducible component C of positive arithmetic genus, then C meets the rest 
ofX in precisely 2 points pi, pi) ifC has genus > 1 then C is PHEL and its contains 2p and pi + p2. 
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Proof, (i) is a special case of Proposition 13.111 (ii) follows from the fact that for any Cartier divisor of 
degree > 2, coc{D) is very ample. Of course, C cannot meet the rest of X in < 2 points. □ 



3. 2-INSEPARABLES AND SEPARATION BY THE CANONICAL SYSTEM 

The purpose of this section is to develop some lemmas needed in the proof of the main result. These 
lemmas concern point separation (very ampleness) properties of the canonical system and some of its 
twists, which are closely related to separating binodes. They also yield short proofs of results of Catanese 1 6| 
and others classifying the 2-inseparable nodal (more generally, Gorenstein) curves whose dualizing sheaf 
is not very ample, and characterization of the limits of hyperelliptic curves (Proposition l3.12T l. 

First some definitions and remarks. 

• A pair of nodes on a nodal curve X is called a binode. A binode {01,62) is properly separating or 
a bisep, for short, if each 0, is nonseparating but X — {01,^2} is disconnected, hence necessarily 
has exactly 2 connected components whose closures are- at this point, arbitrarily- designated the 
left and right parts of 62), denoted 62), rX(0i, ^2); in particular, 9 = {Oi, 62) itself will 
have a left and right preimages denoted l^, ^0. A subcurve of X of the form lX(0), rX(0) for a 
sep or bisep 6 is simply called a side of X. We will call a subset of X a *-sep if it is either a sep or 
bisep. 

• A nodal curve is 2-inseparable if it is not disconnected by removal of any 2 or fewer nodes. 

• The 2-separation of X is the blowing-up of all seps and biseps. The connected components of the 
2-separation are called 2-components of X. 

Lemma 3.1. A 2-component is always inseparable. 

Proof. Suppose a 2-component Y is separable and let be a sep of Y, necessarily not a sep of X. If ^ = 
(01, 62) is a bisep of X contained in Y and £ C lYiO) or C rY(0), 6 is already a sep on X. Otherwise, 
{9i,6) is a bisep on X, contradiction. A sep of X contained in Y leads to a similar contradiction. □ 

Lemma 3.2. Let Xbea 2-inseparable nodal curve and a an effective Cartier divisor of degree >3on it. Then cox{a) 
is very ample. 

Proof. Set a = deg(a),g = pa{X). As h^{a) = g -\- a — 1 , we must prove 

h°{cox{a-h))=g + a-3 

for every length-2 subscheme b of X. Assume first that b is regular. Set L = 0{h — a). This has degree < 
overall and < 2 on each subcurve. By Riemann Roch and Serre duality, what has to be shown is h^{L) = 0. 
If not, let s e H0(L) and C the union of all irreducible components of X on which s is not identically zero. 
Because deg(L) < while deg(L|c) > 0, there must be a connected component Y of the complementary 
curve X — C such that deg(L|Y) < 0, hence s vanishes identically on Y. Because |Y H C| > 3, s\c is a 
regular section of L|c with > 3 zeros, which contradicts deg(L|c) < 2. 

This leaves only the case where b is tangent to a branch at a node 6 of X. Let b : X' ^ Xhe the blowing 
up of 6 and a' = b* (a), of degree a. Then X' is inseparable of arithmetic genus g — 1, and 

h\cvx{a-h))=h\cvx'{a'-p) 
with p the preimage of 6 corresponding to b. What must be proven here is 

H0(Ox'(p-a'))=0. 

This proof is similar to the above, using inseparability of X'. □ 

This result admits a useful partial extension to the inseparable case. First a definition. 
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Definition 3.3. A nodal curve X is said to be 2-inseparable (resp. inseparable) relative to a subset (or divisor or 
subscheme...) a if given a any bisep (resp. sep) 9 ofX, (the support of) a meets both i^X{d) and rX(0). Equivalent 
terminology is that{X, a) is 2-inseparable or inseparable. 

A useful remark about this notion is the following. 

Lemma 3.4. Let Y bea 2-component ofX and 9 a bisep ofX contained in Y. Then Y is inseparable and 2-inseparable 
relative to 6. 

Proof We have seen that Y is inseparable. Let 0' be a 2-sep of Y. If ^ c L^iS!) or c rY(£'), then 0' 
would be a bisep of X itself, against our hypothesis that Y is a 2-component. □ 

The following Lemma on nonpositive bundles will prove useful. 

Lemma 3.5. Let Lbe a nontrivial line bundle of degree < on an inseparable curve X, such L has degree < 2on 
any subcurve ofX and degree < Ion either side of any bisep. Then H^{L) = 0. 

Proof. If s is a nonzero section vanishing somewhere, let Y^, Y2 C X be the union of all irreducible com- 
ponents on which s is (resp. is not) identically zero. Since s is a regular section of L over Y2, it cannot 
have more than 3 zeros there. Also, Yi + 0. Therefore Y\ and Y2 meet in at most, hence exactly, 2 points 
which constitute a bisep. By the assumption on biseps, L has degree at most 1 on Y2, so again we have a 
contradiction. 

□ 

Now we can prove the desired partial extension of Lemma |3^ 

Lemma 3.6. Let X be an inseparable nodal curve and a an effective smoothly supported divisor of degree > 3. 
Assume X is 2-inseparable relative to a. Then cox{a) is very ample. 

Proof. We follow the outline and notations of the proof of Lemma [3.21 We may assume X is 2-separable. 
Consider first the case where b is regular. Then L = (9(b — a) satisfies the hypotheses of Lemma 13.51 
hence H'^(L) = 0, as desired. Now to complete the proof, following the argument of the Lemma 13.21 
it will suffice to consider the case where b is tangent to the left side at a node 9i belonging to a bisep 
{91,62). Using the above notations, we need to show H^{0'^{p — ci')) =0 where p G i,X'{92)- Now X' is 
inseparable, and clearly L = 0'^{p — a') satisfies the hypotheses of Lemma |331 so we are done. □ 

According to Caporaso, an r-vine is a nodal curve of genus g = r — 1 of the form 

(P\pi,...,p,) U '^r). 

Pi ^ qi 
i = 1, r 

An r-vine is said to be an interlace if the pair of pointed P^'s are isomorphic (this notion, but not the term, 
already occur in Catanese's article ||6l). Note that an interlace X admits a 2:1 morphism r] to P^, ramified 
precisely over the nodes and by Riemann-Hurwitz we have 

a;x~ {g-l)r^'^{pt). 

Because we can always assume (pi, p2/ Ps) = {^i, '?2/ ^?,) = (0, 1, 00), the parameter space for r-vines is an 
open subset of (C^^^^)^ and that of r-interlaces is the diagonal. 

An irreducible nodal curve is said to be hyperelliptic if it is obtained from a smooth hjrperelliptic curve 
of genus > 1 by identifying some pairs belonging to the g2, or obtained from a irreducible nodal or 
smooth curve of arithmetic genus 1 by identifying some pairs in a single ^2- The following is an initial 
characterization of 2-inseparable hyperelliptics; the definitive result is Proposition 13.1 1 I below. 
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Lemma 3.7. Let p, q be smooth, not necessarily distinct points on a 2-inseparable curve X. Then p + q imposes 
independent conditions on cox unless either 

(i) X is irreducible hyperelliptic with p-\- q E g\; or 

(ii) X is an interlace and p + q = r]~^ (pt)- 

Proof. By Riemann-Roch, h^{0{p + q)) =2. Assume p, q are in a single irreducible component C. Suppose 
X is reducible and let Yi be a connected component of the complementary curve Y = X — C. The Yi meets 
C in a set a of at least 3 points, and we have an exact sequence 

^ a;Yj ^ a;cuYi ^ '^c(a) -> 

As H^{cvy^) — H^{cvcuYi) (both curves being connected), the map H'^(a;cuYi) — ^ W'^(<^c('j)) is surjective. 
But by the Lemma above, p + q clearly imposes 2 conditions on a;^ (o), therefore also on lox, contradiction. 
Therefore Yi cannot exist and X is irreducible. As |p + <?! induces a degree-2 pencil on the normalization of 
X, or on a genus-1 partial normalization if X is rational, it is easy to see that X is irreducible hyperelliptic. 

If p G C, G D are in different irreducible components, connectedness ensures injectivity of the restric- 
tion maps 

H'{Ox{p + q)) ^ H°{Oc{p))M°{Ox{p + q)) ^ H°{OD{q)). 

Therefore 

h°{Oc{p)),h°{On{q))>2, 

so C, D are smooth rational and there is a 2:1 map CUD — >• P^. We claim C U D = X. If not, let Y be a 
connected component of the complementary curve. Because Y meets C U D in at least 3 points, we may 
assume Y meets C in at least 2 points, therefore coc{Y D C) has nonnegative degree, hence is free. Then 
from surjectivity of 

H°{cvcuY)^H°{coc{CnY)) 

we conclude there is a differential on C U Y not vanishing at p, hence a section of lox zero on D and 
nonvanishing at p. On the other hand because X is inseparable, lox is free so there is a section of cox 
nonvanishing at q. Therefore p, q impose independent conditions on cox, contradiction. 
Therefore X = C U D is an interlace. 

□ 

The above argument proves more generally the following 

Lemma 3.8. Let X, C, Y be 2-inseparable nodal curves with X = C U Y. Then 

(i) the restriction map LL^{cox) — > H^{coc{C n Y)) is surjective; 

(ii) if\Cr\Y\ > 3, then \cox\ is very ample on the interior ofC, i.e. C \ C n Y. 

Next we extend the result of Lemma|3]2]to the case of twisting by a degree-2 divisor: 

Lemma 3.9. Let Xbe a 2-inseparable nodal curve and a an effective Cartier divisor of degree 2 on it. Then cvx{o-) 
is very ample off a, i.e. separates every length-2 scheme except a, unless (X, a) is hyperelliptic. 

Proof. Actually, Proposition 13.111 below- which is proved independently of Lemma 13.91 - already implies 
X is hyperelliptic and this could be used to show o is a hyperelliptic divisor. But we take a different 
tack instead, and follow the outline and notations of the proof of Lemma 13.21 Consider first b, a regular 
length-2 scheme other than a and L = C(b — a), this time of degree 0. If h^{L) = 0, iOx{o) separates b. If 
s Q E H^{L) vanishes nowhere, then C(a) = (9(b) and a is a hyperelliptic divisor by Lemma [3.71 If s 
vanishes somewhere, it is identically zero on some component, and an argument as in the proof of Lemma 
I3.2l applies. This settles the case b regular. The case b irregular is identical to that of Lemma 13^1 □ 

We can partly extend this to the inseparable case: 
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Lemma 3.10. Let Y be an inseparable curve, ip q smooth points such that Y is 2-inseparable relative to p + q. 
Then coyip + q) is very ample ojfp + q, unless Y is 2-inseparable and (Y, p + q) is hyperelliptic. 

Proof. If y is 2-inseparable, the above Lemma applies. Else, p, q belong to different 2-components Y^, Y2. 
It will suffice prove that \co{p + q) |, separates every length-2 scheme b except at p + q. We will focus on 
the b regular case, as the irregular one is handled as in Lemma [3.61 If C(b), hence L = 0{h — p — q) has 
degree < 1 on each component. Lemma [3.51 applies, so we may assume C(b) has degree 2 on a unique 
2-component Z. We may assume p i Z. Let be a bisep on Z with p G i^Y{d),Z c ^Y{9). Then clearly 
any section s e H'^(L) vanishes on i^Y{d), hence on r0 c Z. If G Z, then L has degree 1 on Z but s 
has 2 zeros there, so s vanishes on Z, hence on all of Y. If q i Y, there exists another bisep 6' + d with 
q G iY{6'), Z c •pY{Q')- Then s has 3 zeros on Y, hence vanishes s above. □ 

The following result due to Catanese [6] classifies the hyperelliptic 2-inseparable curves. Given the 
foregoing Lemmas, the proof is short and we will include it. 

Proposition 3.11 (Catanese). Let X be a 2-inseparable nodal curve of arithmetic genus g > 1, such that cox is 
not very ample; then X is either 

(i) irreducible hyperelliptic, or 

(ii) an interlace. 

In either case, the canonical system \tox\ yields a 2:1 morphism onto a rational normal curve in FS~^. 

Proof. Assume X is neither irreducible hyperelliptic nor an interlace. We need to show every length-2 
scheme imposes independent conditions, i.e. is embedded by \cox\- The case of a scheme supported 
at smooth points was considered above. The case of a scheme p + q where p is a node and q p is 
elementary. Next, consider the case of a length-2 scheme ^ supported at a node p. If X' is the blowup 
of p with node preimages pj, p2 then ^ dependent for cox means pi + p2 dependent for cox' (this is true 
whether ^ is regular or not). Note X' is inseparable. If X' is 2-insparable then Lemma [3.71 applies. Else, 
let be a separating binode of X'. If £ is properly contained in a polyseparator (see |J4] below), then 
some connected component of (X')® contains neither pi nor p2, which makes X 2-separable. Therefore 
9_ is already a maximal polyseparator, hence by Lemma [4.31 its sides i^X', rX' are both inseparable and 
again by 2-inseparability of X, each side contains precisely one of pi, p2. Because co ^x'/ <^ ^x' ^re both free 
and H^{cl> ^x')> H°(a; ^^x') C H^{cox') , this implies pi, p2 impose 2 conditions on cox'. □ 

The following characterization of the limits of smooth hyperelliptic curves among 2-inseparables is 
probably not new (compare for instance ||2l, §5.2) but is included because it follows easily from the fore- 
going discussion. 

Proposition 3.12. Let Xbe a 2-inseparable nodal curve of arithmetic genus g > 1. Then the following are equiva- 
lent: 

(i) X is a limit of smooth hyperelliptic curves. 

(ii) The dualizing sheaf cvx is not very ample. 
(Hi) X is irreducible hyperelliptic or an interlace. 

Proof. The equivalence of (ii) and (iii) is just Catanese's result (Proposition 13. lit . That (i) implies (ii) is 
trivial: if a smooth hyperelliptic curve X specializes to a stable curve Xq, then a general point p G Xq 
will be contained in a length-2 scheme imposing 1 condition on coxg, so coxq is not even birationally very 
ample. It remains to prove that (iii) implies (i). This is basically a folklore dimension counting argument 
that we will give in the case of an interlace, as that of an irreducible curve is similar. Thus let Xq be an 
interlace, which we may assume corresponds to 2 identical {g + 1) -pointed P^'s of the form 

(P},0,l,OO,pi,...,p^_2) ~ (P^,0,l,OO,(^i,...,£J^_2). 
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Let X/B be a versal deformation of Xq. Let Bi c B denote the locus of {g + 1) -nodal curves, which may 
be identified locally with (C^~^)^ x T for a polydisk T, with the 2{g — 2) parameters corresponding to 
pi, cjg-2- On B, Bi is locally defined by (g + 1) equations corresponding to the nodes, say ui, m^+i- 
Let p be a general point of P^, and let p' G Pj C Xq, p" eW\cXq be the points corresponding to p. Then 
in a neighborhood of (p', p") we may identify X| with (C^"^)^ x T. 

Now let E be the Hodge bundle tt* (cox/b) pulled back to X| and consider the evaluation map 

(p:E^ pKcox/b) ffi pK-^x/b)- 

At {p', p"), (p has rank 1 hence in a neighborhood of (p', p") on Xg, the degeneracy locus D of (pis defined 
by ^ — 1 equations, say fi, fg-i, not necessarily forming a regular sequence. On the other hand, consider 
the restriction of (p over X| , identified with {C^-'^)^ x T. There, the de generacy locus of (p corresponds 
to hyperelliptic {g + l)-nodal curves plus member of the g2, and by Proposition l3 .111 this can be identified 
with C^~^ x T sitting diagonally in (C'?^^)^ x T, hence is still defined by a regular sequence of length 
g — 1. It follows that fi, ...,fg_i forms a regular sequence mod (uj, M^+i), i.e. /i, Ui, u^^i 

together form a regular sequence, hence no m, is a zero-divisor modulo /j, i.e. D has no compo- 

nent contained in any boundary divisor corresponding to a node of Xq. Therefore (p', p") is a limit of 
hyperelliptic divisors on a smooth hyperelliptic curve. 

□ 

Remark 3.13. Via the theory of admissible covers [11] and its generalizations as in |]8| and fflOj, the limits 
of hyperelliptics can also be characterized among the stable 2-inseparables as those admitting a 2:1 map 
to P^. A drawback of this characterization is that it does not readily yield defining equations for the 
hyperelliptic locus. By contrast, the above result readily yields such equations, more precisely, equations 
for the locus of pairs (hyperelliptic curve, fibre of hyperelliptic map) as a degeneracy locus (always among 
2-inseparables). 

Now as soon as one leaves the realm of 2-inseparable curves, birational non-very ampleness of the 
canonical system is no longer a good notion of hyperellipticity: as the example below shows, the canon- 
ical map can have different degrees (1 or 2) over different components of its image, and such curves 
cannot be limits of smooth hyperelliptics. This is precisely the motivation for developing the notion of 
the 'sepcanonical system' the will occupy us in subsequent sections. One of the main results of the pa- 
per fl3J will show that the limits of hyperelliptics are precisely the curves whose sepcanonical system is 
not birationally very ample. 

Example 3.14. Let be a separating binode on a nodal curve X, so that 

( lX(0), L^i + L^2) is a smooth hyperelliptic pair while rX is non- hyperelliptic. Then ojx is not essen- 
tially very ample but X cannot be a fibre of a family with smooth total space and general fibre a smooth 
hyperelliptic curve X'. This will follow easily from the results of the later sections but in an elementary 
fashion can be seen from the fact that the locus of hyperelliptic divisors (pairs not separated by the canon- 
ical system) is a divisor on (X') which would extend to a divisor on the relative Hilbert scheme of the 
family (which can be assumed smooth); however, this locus meets the component ( rX)(^) of the Hilbert 
scheme X^l in the isolated point 6i+d2- 

A similar can be made with 9 replaced by a separating node, hyperelliptic only on one side. 

4. POLYSEPARATORS 

This section is essentially trivial in nature. Its purpose of this section is to collect for future reference 
some elementary, and probably well-known (compare [4J, |5|), combinatorial remarks about nondisjoint 
collections of separating binodes on a curve. We begin with a few definitions. 

Definition 4.1. Fix an inseparable nodal curve X. 
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(i) A polyseparator of degree n on X is a collection of nodes {6i, ...,9„),n > 2 with the property that any 
distinct pair of nodes 6,6' e & is a separating binode. 

(ii) is a proper polyseparator ifn>3. 

(Hi) A binode or polyseparator is maximal if it is not properly contained in a polyseparator. 
(iv) We associate to a polyseparator a graph G(0) having as vertices the connected components of the 
separation or blowup X® = b^X and as edge-set 0. 

Definition 4.2. (i) On a general nodal curve X, we define a polyseparator as a polyseparator on one of its 
inseparable components. 

(ii) X is said to be of semicompact type (or 'polyunseparated') if it has no proper polyseparators. 

Lemma 4.3. Let & be a polyseparator of degree n on an inseparable curve X. Then (i) G(0) is a simple n-gon 
and, with suitable notation, 

n 

X — [j RX{6i,6i^i),6„^i :— 6i; 

(ii) given a node ^ 0, U {9} is a polyseparator ijfd is a separating node on the unique rX(0,, containing 
it.. 

Proof. Induction on w > 2. With (i) being obvious for n = 2, we first prove (i) implies (ii) for given 
n. Notations as above, suppose is a node on rX{6i,02) separating it in 2 cormected components L,R. 
Because X is inseparable, precisely one of di, 02 is on each of L, jR and we may assume di & L,d2 & R. But 
then clearly the separation 

X^'^i = [RU U RX(0y,Vl)]U[U rX(0;,Vi)UL] 

j=l j=i 

SO that {6, 6i) is a separating binode. Conversely, suppose is a nonseparating node on rX(0i, 62). Then 
has just 2 connected components and becomes connected when the two preimages of 62 are iden- 
tified, i.e. X^'^i is connected, so (6, 61) is nonseparating and U {6} is not a polyseparator. 

To complete the proof it suffices to show that (i) and (ii) for given n implies (i) for n + 1. So suppose 
= (01,..., 0„+i) is a polyseparator of degree n + 1. Then {6i,...,6„) is a polyseparator and we may 
assume the above notations apply. Using (ii) and shifting cyclically, we may assume 0„+i is a separating 
node on rX{6„, 61) and inseparability of X again implies that 0„+i has exactly one of 6„, 9i or either of its 
sides in rX(0„, 9i). This shows G(0) is an (n + l)-gon. □ 

A notation as above is called a cyclic arrangement for the polyseparator 0. 

Lemma 4.4. Any polyseparator on an inseparable curve is contained in a unique maximal polyseparator. 

Proof. Let be a polyseparator and use a cyclic arrangement as in the previous lemma. Let M(0) be the 
union of and the separating nodes on rX(0i, 0/+i). By the Lemma, any polyseparator containing is 
contained in M(0). Therefore it suffices to prove M(0) is a polyseparator. The proof is by induction on 
m(0) := |M(0) \ 0|. If m(0) > 0, pick any 9 G M(0) \ and let 0' = U {9}. To finish the mduction 
it suffices to prove M(0') = M(0). With no loss of generality, we may assume 0' = {9-[, 0„+i) is a 
cyclic arrangement as above. If a G M(0'), we may assume a is a separating node on rX{9„,9„+i). By 
inseparability of X again, a has exactly one of 9„,6„^i on either side in rX{9„,9„+i), which makes it a 
separating node on rX{9„, 61), so a € M(0). Conversely, if a G M(0), we may assume a is a separating 
node on y = rX{9„,6i) with 9n E i^Y{a),9i e rY{<x-)- But then if 9n+i e then a is a separating 

node on rX(0„+i,0i), while if 0« e rY (a), then a is a separating node on RX{9n,6n+i). Therefore in 
either case a G M{& ). a 

Lemma 4.5. Any two maximal polyseparators on an inseparable nodal curve are disjoint. 
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Proof. Let — {6i, ...,6„) be a maximal polyseparator, cyclically arranged, and 9 another node, say 6 G 
rX(0;, If 6 separates rX(6,-, ^i+i), then G by maximality. If not, then clearly X^i'^ is connected 
for all ; (we can go around the circle starting from 9j and avoiding 9). Therefore, 6j cannot be in any 
polyseparator not contained in 0, so cannot be in any other maximal polyseparator. This proves the 
required disjointness. 

□ 

We summarize the above results as follows. 

Proposition 4.6. Let X be an inseparable nodal curve. Then there is a collection of disjoint sets of nodes on X called 
maximal poly separators, with the property that a given binode 9 is properly separating iff 9 is contained in some 
maximal polyseparator. 

A (oriented) bisep is said to be adjacent (or right-adjacent ) on X if rX(^) is inseparable. This termi- 
nology is justified by the following result, which follows easily from the above discussion: 

Corollary 4.7. {9\, 92) is adjacent on X iff the unique maximal polyseparator containing it has the form, in cyclic 

arrangement,® = {9^,92, ...,9m), m > 2. 

Lemma 4.8. In a curve of semi-compact type, any two separating binodes lie entirely to one side of each other. 

Proof. Let 9^ 9' he separating binodes of X, and assume for contradiction that 9[ G l^{9), ^2 G R^ii.)- 
Then b0i^ lX{9), bgi^ rX{9) are cormected and their images in bQiX clearly meet, so b^iX is connected, con- 
tradiction. □ 

The following two results collect some elementary properties of graphs of curves and of curves built 
up from 2-inseparable hyperelliptics, respectively. All the required proofs are easy enough to warrant 
omission. 

Proposition-definition 4.9. Let Xbe a nodal curve. 

(i) The 2-separation tree G2(X) is the dual graph having as edges the separating nodes and all maximal 

(i.e. not contained in a proper polyseparator) properly separating binodes, and as vertices the connected 
components of the blowup of all the latter nodes and binodes. 

(ii) This graph is a tree. 

For a nodal curve X, we will denote by S(X) the set of all its seps and biseps. Recall from the beginning 

of the previous section that a A 2-separation component (or 2-component) of X is by definition a connected 
component of the blowup of X in S(X), and that a 2-component is always inseparable. 

Proposition-definition 4.10. (i) Any 2-component Y is endowed with a collection of smooth singletons or 
'unimarks' coming from separating nodes of X, and smooth pairs or 'bimarks' coming from separating 
binodes. Unless otherwise mentioned, these are always oriented with Y on the left, 
(ii) A sep 9 is said to be locally left-hyperelliptic if for the 2-component Y to the left of 9, (Y, l^) is hy- 
perelliptic; ditto for locally right- and bilaterally (left + right) hyperelliptic; ditto for bisep in place of 
sep. 

(Hi) a locally left or right-hyperelliptic bisep is automatically maximal. 

(iv) A stable curve X is said to be hj^erelliptic if it is of semicompact type and every edge o/G2(X) is 
locally bilaterally hyperelliptic; given X, it is is hyperelliptic iff every sep and bisep is locally bilaterally 
hyperelliptic. 

(v) For a subset C S{X), Xis said to be hyperelliptic relative to if every 9 e & is locally bilaterally 

hyperelliptic. 

(vi) A sep 9 is said to be left hyperelliptic if ( lX{9), i^9) is hyperelliptic; ditto for right and bilateral; ditto 
for bisep; 
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(vii) 6 is left hyperelliptic iff it is locally left hyperelliptic and every sep and bisep strictly to the left of 6 is locally 
bilaterally hyperelliptic. 

(viii) 9 is said to be left hyperelliptic relative to it is locally left hyperelliptic and every sep and bisep in 
strictly to the left of 6 is locally bilaterally hyperelliptic; 

(ix) A connected nodal curve X is said to be hyperelliptic if its stable model is. 

5. Residue Lemma 

The definition of the sepcanonical system to be given in |j6] is motivated by the following elementary 
Lemma about variations of twists of the canonical system in a family To introduce the Lemma somewhat 
informally, consider a degeneration of curves X / B with smooth total space over the disk, and let D be a 
sum of components of the special fibre, and let s, : B X be sections disjoint from D, i = 1, n. Then the 
bimdle cox^ ® (^(D) ® C'xo)(ES)(0)) is a limit of <j^x,,(ES)(fe)) on the general fibre. What is the condition 
that a section ocq of the former bundle lift to a section a;, of the latter bimdle ? An obvious necessary 
condition is I]1^6Ss.(g)(ao) = (because the analogous sum for a;, vanishes by the Residue Theorem). The 
Lemma below says that under some mild hypotheses, this condition is also sufficient. 

First some terminology. For a family of curves tt : X — > B, an etale multisection is an effective divisor 
(J C X that is finite flat and etale over B and disjoint from the singular locus of X/B. B may be a point, in 
which case an etale multisection is just a reduced divisor disjoint from the singular locus. For such a and 
an effecive divisor A disjoint from a we have the residue map 

Res : n^{cox/Bi.A + no-)) 7r*(C^). 

Composing with the trace map, we get a 'sum of residues' map 

^Res : n4cvx/BiA + ncr)) Og. 

cr 

When B is local, we will identify tt* with H". 

Lemma 5.1 (Residue Lemma). Suppose given 

• TT : X — > B, fl proper flat family of nodal curves of arithmetic genus g, with irreducible generic flbre, over 
an integral local scheme; 

• Bg C B, fl closed subscheme (the 'boundary') with ttq : Xg = X x g Bq — > Bg the corresponding subfamily; 

• pairwise disjoint etale multisections cr,-, i = 1, m ofX/ B, identifled with the associated relative divisors; 

• an effective Cartier divisor D C X, disjoint from md from the generic flbre, such that, setting 
Oxo(Dg) = Ox{D) iSi Oxg, ive have that the direct image sheaf TZo^{cvxo/ Boi^o)) is free of rank g over 
Bg ('constant sections hypothesis'); 

• where cr,- g = cr,- n Xg, an element ag e H° (Xg, o^Xq/Bo (^'o + E "/"^/^o) )/ > 1- 

Then: 

m 

(i) ag z's a sum Yj i of sections 
1=0 

ao,i e H°(Xg,cJxo/Bo(D + n,ng)) c H°(Xg,a;xo/Bo(D + Ew)) 
iffoiQ satisfles the residue conditions : 

(51) ^Res(ag) = e Cbq,z = l,...,m. 

Moreover the set of sections satisfying the residue conditions iS.l^ is a free Osg- module of rank 

m 

g+Ylni deg(c7,- g) - m. 

1=1 
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(ii) If the residue conditions dS.ll l are satisfied, then lifts to an element a of 
H^{cvx/b{D + LniO-i)) satisfying 

^Res(a) = 0eOB,i = 

Remarks 5.2. (i) Because X] and D are disjoint, X admits an open affine cover 

{Uo:=X\J2cri,Ui), 

where Ui is an affine neighborhood of J^Cj contained in X \ D. We can use this cover to compute coho- 
mology and over UqCiUi, naturally identify cox/B and lox/b{^) and compute residues for the latter sheaf 
using this identification. 

(ii) In applications, D will usually be a sum of boundary components, i.e. components of Xg^. 

Proof of Lemma, (i): By the constant sections hypothesis, 7r*(a;x/B(C')) is free of rank g and compati- 
ble with base-change. Hence R^n^{cvx/BiD)) is free of rank 1 and compatible with base-change, more 
precisely the map R^7T*(a;x/B) ^ ^^^*('^x/b(C')) is an isomorphism and remains one after any base- 
change. Let A be the set of sections expressible as a sum J] ag , as above, and let B be the set of sections 
tXQ satisfying the residue conditions 15.11 Now for each consider the long cohomology sequence of 

^Xo/Bo{Do) a;X(,/Bo(Do + w/t^,-o) ^ ^Xo/Boi^o) C'n;cr,.(,(n,-(7,-,o) ^ 0. 

The Dq twist is trivial on the third term because D is disjoint from cr,. This shows that 
^^°(^Xo/Bo(^o + niCTi^o)) maps surjectively to the set of 'polar parts' in H°{cvxg/BoiDo) ® C'„;cr,.(,(«/cr,,o)) 
satisfying the fth residue condition in 15.11 This implies firstly that A C B, i.e. 'only if holds. Then, 
considering the maps 

ACBC H°{cox„/B,iDo + J2^iCri,o)) ^ hO((9„,.,„(^«,c7,-,o)) 

it follows that A, hence B, map onto the submodule C of H'^(C«,i7,o(E"('^(,o)) consisting of polar parts 
satisfying all the residue conditions 15. 1 1 for ; = 1, m, which is obviously free and cofree of corank m. 
Because we have an exact sequence 

O^HO(a;x„/Bo(Do)) ^A^C^O 

and likewise for B in place of A, it follows that A = B is free of the required dimension. 

(ii) By (i), it suffices to consider the case m = 1 and we will omit the index. Consider the exact diagram 

H^{co{D)) -> H^{co{D + na)) H^{u}{na) ® Ona -> H^{w{D)) 

Here the rightmost horizontal maps are just residue. In fact the entire rightmost square is purely local and 
it it easy to check that the kernel of the upper right horizontal map- which is a surjection of locally free 
sheaves- goes surjectively to that to the lower right horizontal map. This proves our assertion. □ 

Remark 5.3. The residue condition is automatically satisified on an irreducible fibre but not on a reducible 
one. 

Example 5 A. Hereisanillustrationof how the Residue Lemma is applied. Let Xq = i^XqUq^ ^Xq Ug^ rXq 
be a 3-component 2-sep nodal curve, varying in a 2-parameter smoothing family X/B, with divisors 3i, 82 
where 61,62 persist, respectively. Let lD = i^X{6i), a divisor on X. Then 

^x/b{D) I j^x = ^x(2 R01 + lOi)- 
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We claim that a necessary and sufficient condition for a section a of the latter sheaf to extend locally to a 
section of 7T*(a;x/B(^-')) is that a should have zero residue, i.e. trivial pole, at 162- This of course implies 
a has zero residue at as well. 

We will prove sufficiency of the condition as necessity is similar and simpler. To this end, note first that 
by freeness of cv L^i' ^ extends to a section of cvx/b \ lXqU mXq- Then by the Residue Lemma, this 

extends to a section of 

'^lX(02)(^+ l^z) = a;x/B(D)| j^x(92) 
over 82 with zero residue on id2- Then we glue this to any section of 

^x/b\ rX(02) = '^x/b(D)| jjX(e2) 
(which automatically has zero residue on ^62) arid use the Residue Lemma again to get a section of 
a;x/B- □ 

The foregoing example will be generalized in Theorem |7.5l below. 

6. Sepcanonical system 

The purpose of this section is first to define intrinsically an object on a reducible curve Xq, viz. a col- 
lection of linear systems on its 2-components, collectively called the sepcanonical system of Xg. This will 
later be shown to coincide with a certain collection of limits of the canonical system, i.e. specializations of 
its twists by certain components of the special fibre. This construction is nontrivial when the special fibre 
is 2-separable. The sepcanonical system depends on some additional data on the curve called azimuth 
(essentially, a smoothing direction at each bisep). 

Definition 6.1. (i) lf{p\, pi) is a pair of distinct smooth points on a curve X, the azimuth space at [pi, P2) 
is P(TpjX ® Tp^X); an azimuth ^ at (pi, P2) is an element of the azimuth space. An azimuth is singular 
if it equals [1,0] or [0,1], regular otherwise. An azimuthal pair {{p\,p2),0 consists of a point-pair 
(Pi/ Pi) P^us an azimuth ^, as above, 
(ii) For a bisep (properly separating binode) 6 = {61,62), a left (right) azimuth is an azimuth on its left (right) 
preimage , i.e. an element e P( l'/'Si ® \Jpe2) (^<^sp. ^l, e P( Rip^j © ^.^62))' ^here ^xpg. = 
T\e, etc . 

(Hi) For a bisep 6 = {61,62), a middle azimuth is an element 

is regular if ^l, = [a, b], a,b ^ 0. An azimuth is a triple = ( l^, j^^, consisting of a left, 
middle and right azimuth, is said to be (internally) compatible if 

(iv) An azimuthal (resp. middle azimuthal curve) is a nodal curve together with a choice of compatible 
azimuth (resp. middle azimuth) at each separating binode. 

Definition 6.2. An azimuthal marking ^ = {p.; {q., q' .)) on a curve Y consists of. 

(i) a collection of smooth points (unimarks) pi e Y, together with a designation of each as either ' co-hyperelliptic' , 
with attached 'multiplicity' n{^, p) = 1, or 'non co-hyperelliptic , with attached multiplicity n{^, p) = 3; 

(ii) a collection of smooth pairs (bimarks) {qi, q\), together with a designation of some of them as 'co-hyperelliptic', 
and a choice of regular azimuth l,{qi, q\) (only) at those. 

Remarks 6.3. (i) The co-hyperelliptic property of a uni- or bimark and the 'multiplicities' above are, 
from Y's perspective, completely arbitrary, and in particular is unrelated to being a W-point or 
hyperelliptic divisor on Y itself. In applications, there will be another curve attached to Y at those 
points, and the data will be related to hyperellipticity of the other curve. 
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(ii) In an azimuth ^, any two of l^, if not both singular, uniquely determine a third so that 
the triple is a compatible azimuth. 

(iii) If pi ^ p2 and pi + p2 belongs to the tmique on a hyperelliptic curve X, there is uniquely 
determined on pi + p2 a hyperelliptic azimuth, automatically regular. 

(iv) In particular, given a regular middle azimuth y^l, on a right-h5^erelliptic bisep dj there a uniquely 
determined left azimuth on 0, compatible with and the hyperelliptic right azimuth. 

(v) Given a bisep 9_ = 62) as above on a curve Xq and a smoothing X/ B of it, then an element v of 
the projectivized normal space to 89^ n de^ at induces a middle azimuth at ^ ]y[^ is regular 
iff V corresponds to an infinitesimal smoothing at both 9\ and 02- 

Now the sepcanonical system associated to a middle- azimuthal curve X, which we are going to define, 
will consist of 

• a sepcanonical bundle , a line bundle on the 2-separation of X, i.e. the disjoint imion ]J X,- of 
the 2-components; 

• a linear subsystem |a;x|''^P C called the sepcanonical system. 

More generally, we will define a 0-sepcanonical system for any collection of seps and biseps so that the 
ordinary sepcanonical system corresponds to the case where consists of all seps and biseps. 

Definition 6.4. Let 9 be an oriented *-sep (either sep or bisep) on a nodal curve X, @ a collection of seps and 
middle-azimuthal biseps. 

• 9 is said to be right-hyperellptic relative to if 

(i) the pair ( rX(6), r6) is hyperelliptic; 

(ii) every element of (0 \ 0) n ^X{9) is hyperelliptic as a sep or middle-azimuthal bisep on rX(0). 

• if9_ e & is a right-hyperelliptic bisep relative to 0, the induced left azimuth on 6 is by definition the one 
induced by the given middle azimuth at 9 and the hyperelliptic azimuth on r0. 

Definition 6.5. Let (X, ^0) ^ stable marked curve, a collection of seps 9i and middle-azimuthal biseps 

mC^(^i)' ^ component of the separation X®. 
Then the induced azimuthal marking ^(X,0) on Y is defined as comprised of the following unimarks and 
bimarks. Assume all seps and biseps of& are oriented to have Y on their left. 

(i) Unimarks: those inherited from ^q, plus unimarks p = i9i,9i e 0, designated as co-hyperelliptic iff 9i is 
right-hyperelliptic relative to 0; the induced multiplicity is denoted ^n{X,&,p) or ^n{X,p) if @ contains all 
*-seps on X. 

(ii) Bimarks: those inherited from ^q, plus bimarks {{qj, q'^) = l^,}, 0,- G 0, designated as co-hyperelliptic iff9_i 
is right-hyperelliptic relative to 0, and in that case endowed with the induced azimuth. 

Definition 6.6. Let Ybea curve with an azimuthal marking f — (p., {q., q'.), [,.). Then define the following items 
on Y: 

• The sepcanonical twist: 

(6.1) t(Y,^)= YL «(?'P)P+ E ^ili + H'i) 

marks p bimarks (qi,q[) 

• The sepcanonical bundle: 

(6.2) a;(y,0 =^y(T(X,y)) 

• The sepcanonical system \cv\^'^P {Y , ^) C |a;(Y,^)| is the subsystem defined by the following conditions: 

(i) Residue conditions: 

Resp;(a) = 0,Vz; 
Res,;(a) + Res^/(a) = 0,Vf. 
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(ii) Azimuthal conditions corresponding to the given azimuth at all co-hyperelliptic bimarks. 

• When ^ = ^(X,0) and Y is a component ofX®, the same objects will be denoted by [o^xIq^Iy ^fc. and 
the collection of these for different components Y is called the sepcanonical system of X associated to 0, 
denoted lo^xl©^- 

• If a is a smoothly supported effective divisor of degree > 2 on X, we define toxi^^y^^ and \cvx{<^)\^'^^ 
analogously, pretending all seps and biseps are non-hyperelliptic and imposing residue conditions only. 

• When = Edge(G2(X)), will be omitted from the notation and the corresponding system \cvx\^^^ 
will be called the (absolute) sepcanonical system of (the middle-azimuthal curve) X. 

Remarks 6.7. (i) Note the natural inclusion 

^x^|YCa;x|y(2EL0,+E('?>- + '?;)) 

(ii) It follows from Lemma |5A] that an element of the sepcanonical system on Y is a sum of sections of 
coy{ Rn{Oi) i^Oj) and cvyi^icji + q'^) for the various i. 

(iii) The definition of a;x(a)®^P and |a;x(i) 1^*^^ is certainly not the 'correct' one in general, but it is good 
enough for the few cases we need. 

(iv) The sepcanonical system \cox\'^'^^ is highly non-local in the sense that its restriction on one 2- 
component Y depends on the nature of other 2-components (as well as on the azimuthal structure). For 
example, we allow a pole of order 3 at p = on Y, for a sep 6, when p is not co-hyperelliptic, i.e. when 
the other (right) side Z of in X is non-hyperelliptic at r0 because coz will then have a section with a 
simple zero at r0 to match with a differential on Y with pole of order 3 at p. 

Remark 6.8. We can extend the definition of sepcanonical system to the case where X is semistable. (This 
is not essential, because a given family of curves can always be replaced with one with only stable fibres, 
albeit at the cost of making the total space singular.) Then the stable model X of X is obtained by con- 
tracting a number of 'bridges', i.e. maximal rational chains each contracting to a node p of X. The bridge 
is said to be separating or not depending on p. We firstly define |a;x|^'^I' to coincide with the pullback 
of |a;x|'^'^P over all components not contained in bridges. Next, we will give a recipe for extending this 
over the bridges. The recipe may appear arbitrary, but it will be justified later in the discussion of mod- 
ified Brill-Noether maps. Thus consider a component C of a bridge B. Then C is naturally a 2-pointed 
and may be identified with (P^, 0, oo). If B is a separating bridge, with sides ( i^X, ^p), ( rX, ^p), we 
define |(i^x^'^'')cl to be the linear system generated by the following differentials (also given in terms of 
homogeneous polynomials, as subsystem of the appropriate 0{n), identifying o^pi = O{2oo)): 

(i) dx, dx/x^; [Xq, X\], if ( lX, ^p), ( rX, ^p) both hyper elliptic; 

(ii) xdx, dx, dx/ x^; [Xq, XgXj, X^], if only ( lX, ^p) hyperelliptic; 

(iii) dx, dx/x^, dx/ y?; [Xq, XgX^ X^], if only ( rX, j^p) hyperelliptic; 

(iv) xdx, dx, dx / x^, dx / x^; [Xq, X'^Xi, XqX^, Xf], if neither side hyperelliptic 

(these define respectively a bi-ramified double line, a cusp-flex/flex-cusp plane cubic, and a biflex skew 
quartic). Note that the linear systems in question are uniquely determined by their degree (as specified by 
the lw(0), Rn{6) fimctions) plus the residue condition. Thus, the above apparently arbitrary definition is 
forced. 

If C is a component of a nonseparating bridge B, with end points ^p, ^p E X, we define | (<^x^'^^)c 
analogous to case (i) above if ^p, jjp is a hyperelliptic divisor on the complementary curve X' = X — B; 
and analogous to case (iv) above, otherwise. 

We note that with this definition, it remains that |a;x|^'^P defines a g2 on each 2-component, when X is 
semistable hyperelliptic. 

Remark 6.9. Some inductive arguments may involve the case of a 'shoot' C, i.e. a nonsingular rational 
curve meeting the rest of X in a single point p. Such a shoot, if part of a larger semistable curve X' D X, 
will also occur on a bridge on some other subcurve X" c X', hence dealt with as above. 
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The following Theorem identifies the sepcanonical system as the limit value of the direct image of 
a certain twist of the relative canonical bundle. It should be mentioned that for the Eisenbud-Harris 
limit series associated to the canonical, some analogous results, especially for curves comprised of two 
components meeting in points in generic position, were obtained by Esteves-Medeiros ( [9J, §5). 

Now the rationale for our particular choice of twist and limit is that it is the 'best of the worst'; namely, 
its restriction on a given 2-component Y is the 'best behaved'- from Y's viewpoint- among those limits 
of the canonical system which are available in all cases, including the 'worst' case, when components 
across from Y are hjrperelliptic; or, what amounts to the same thing, this limit does not jump as X varies, 
preserving Y. To make this precise, one has to turn the binode boundary loci into divisors, which in 
general involves the 'azimuthal modification' of a given family which we now define (see [13], §6 for 
more details). 

Let X/B be a family of nodal curves, assumed versal near the special fibre Xq. Each bisep £ of Xq 
corresponds to a regular codimension-2 subvariety dg. Given a collection ©2 of disjoint biseps, the corre- 
sponding loci dg are transverse. Let B(02) denote their joint blowup in any order, which is independent of 
the order. Moreover assume each G ©2 is oriented, so there is a choice of components i,X{6_), rX(0) of 
Xg^ . Then for each 6_ E ©2, the inverse image of 'lX{6_) on the base-changed family Xg^Q^^ is a non-Cartier 
divisor. We let Xg^^^j be their common blowup, called the left azimuthal modification associated to ©2. 
Let L^B{02) (--^ associated Cartier divisor on Xg^^^) . 

Given this notion, the result is as follows . 

Theorem 6.10. Let the following be given: 

TT : X ^ B, 

a family of nodal or smooth curves with irreducible general fibre, assumed versal near a fibre Xq; 

• ©1 , fl set ofseps, ©2 a set of maximal, regularly middle-azimuthal biseps on Xq; 

• with = ©1 U ©2, a connected component Y of the separation X®. Assume each 6i e ©1 and 9j E ©2 
is oriented having Y on its left. 

Let 

n':X' = Xi,,g(0^) ^ B' = B(©2) 

be the associated left azimuthal modification and 0' E B' a preimage of corresponding to the given collection of 
regular middle azimuths on ©2. 
Then: a section 

aoEH'{coy{3 ^ ^9, +2 ^ {^0,))) 
extends near 0' E B' toa local section 

(63) ae<(a;x7B'(-2 E lX'{0)- ^ lX'(0))) 

if and only if 

(6.4) oiQ E |a;x|0''|Y. 

Proof. The proof is in essence an application of the Residue Lemma l5Al We first prove sufficiency of (|6.4| |, 
i.e. 'if. Denote the line bundle in I I6.3D on X' by co'. Because l^'(^) + R^'(^) = for sriy *-sep 9, co' is 
isomorphic locally over B' to 

a;x7B'(+2 E E 
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Now the latter sheaf contains an analogous one where ©i, ©2 are respectively replaced by ©j, ©2 which 
refer to the *-seps that meet Y. Therefore, locally over B', oj' contains a subsheaf 

cv'o^cvx'/B'{-2 E lX'(0)- E lX'(0)) 

Moreover, lo'q is isomorphic to to' near Y. Therefore in the proof of sufficiency, we shall henceforth assume 
all members of ©1, ©2 meet Y. 

Next a remark. Let (Z,z) = ( j{Xo(0;), r^i) for some i, i.e. the side of Xg across 0; from Y. Then 
\coz\ is always free at z and coz{—2.) is free at z iff (Z,z) is non-hyperelliptic. Similarly, if (Z,zi,Z2) = 
( rXo(^,)/ r^/)/ then the image of \toz\ in coz ® ^^21,22 is the hyperelliptic azimuth if (Z,zi,Z2) is hyper- 
elliptic and the whole 2-dimensional space otherwise. From this it is easy to see that the azimuthal con- 
ditions, together with the lowered pole order condition ( 2 rather than 3) at the hyperelliptic unimarks, 
cf . Definition 16.41 are necessary and sufficient for ocq to extend to the entire special fibre as a section of 
coxq{J1 Rn{Oi) rXo(0,) +2Y^ RXo{Oj). Now we use an induction onm = |©|. The case m = 1 is trivial. 
Pick any 9 E & (sep or bisep), let and work over B(© \ 6). Using Lemma |5T] for Y U i^X'{9), we can 
deform ag to a differential oci on a component Yf of a nearby fibre Xj smoothing out exactly 61 and keep- 
ing @\6 intact, where cci satisfies similar residue conditions, ;« — 1 in number, as well as appropriate 
azimuthal conditions (the fact that the latter can be imposed follows from an argument similar to the 'ar- 
bitrary residues summing to zero' argument in the proof of Lemma f5.ll . Then by induction ai, hence kq, 
extends to the general fibre. This completes the proof of sufficiency. 

We now prove necessity of l|6.4t , and accordingly drop the hypothesis that all members of © meet Y. 
We fix a bisep 6_ c Y that is right-hyperelliptic relative to © and establish that azimuthal and residue 
conditions for ag at ^0; other cases are similar or simpler. 

Consider a *-sep 0' G © on rXq, (6) that is extremal relative to Y, i.e. such that there is no element of © 
strictly right of 6'. Thus rXq, {9') is a component of ( rXq,)®. Let X[ be the unique component of ( rXq,)® 
containing the left side \^6'. Because our family is versal, there is a nearby fibre X'^/B' where 6' deforms to 
a bisep 9^, ( rXq,(0'), r0') survives intact, while i^Xq,{9'i) smooths out. Because a extends to this fibre, an 
argument as above shows that the residue and azimuthal conditions hold at ]^9f. Hence by specialization, 
they also hold at i^9'. Because this is true for every extremal 9', it follows that the restrictions of a on 
X[ belongs to the linear system \a>\^'^^ {X[, ^) where ^ is the azimuthal marking on X[ induced by the 
collection of the extemal 9' . Now the latter system is composite with the canonical system of Xj, hence it 
fails to separate any hyperelliptic pair on X[ and is ramified at every W-point (see Lemma [7!2|l . Therefore 
we can continue and 'peel off Xj just like rXq, (0'): the azimuthal (and residue) conditions are satisfied 
on the left of every bisep on X^ and similarly for seps. Continuing in this way, we eventually reach 9_ on 
Y, so the conditions are satisfied there as well. 

□ 

7. Semistable hyperelliptics: general case 

A stable, say semicompact-type, curve X may be viewed as built up from 2-inseparable 'atoms'; as 
such, the canonical system of X behaves in an entirely heterogeneous manner: it can be birational on some 
atoms but not on others. This is not so for the sepcanonical system. Indeed a remarkable property of 
the sepcanonical system (on a middle-azimuthal stable curve), to be established here, is that its mapping 
behavior is quite homogeneous: either it is 'essentially very ample' (in particular birational on every atom); 
or else the curve is hyperelliptic in the sense defined above, i.e. comprised of 2-inseparable hyperelliptic 
atoms in a tree-like arrangement (and in particular is of semicompact type), and the sepcanonical system 
yields a 2:1 mapping on every atom. In particular, different 2-components influence the behavior of the 
sepcanonical system on each other. This property is part of the main result of this section (Theorem l7.5t . 
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A stable middle-azimuthal curve is said to be hyperelliptic (as such) if it is hyperelliptic as ordinary 
curve and if moreover the given middle azimuth ij[^{0_) at every bisep 0, necessarily hyperelliptic, is 
the hyperelliptic middle azimuth, i.e. the one determined by the hyperelliptic left and right azimuths at 
6_. A semistable curve is pseudo-hyperelliptic if its stable contraction is. Our purpose here is to extend 
the characterization of 2-inseperable hyperelliptics in terms of very ampleness of the canonical system 
to the case of general nodal middle-azimuthal curves, where the canonical system is replaced by the 
sepcanonical one. 

We begin with an elementary Lemma on the effect of azimuthal constraints on very ampleness. 

Lemma 7.1. Let Y be an inseparable nodal curve, p ^ q E Y a pair of smooth points such that Y is 2-inseparable 
relative top + q, and I, an azimuth at {p, q). Then the (,-constrained subsystem 0/ |a;y (2p + 2q) \ is essentially very 
ample unless Y is 2-inseparable hyperelliptic, p + q is a hyperelliptic divisor and I, is the hyperelliptic azimuth. 

Proof. If y is not 2-inseparable, then toyip + q) is already essentially very ample by Lemma [3. 101 hence so 
is the constrained subsystem in question. If Y is 2-inseparable and p + q\s not a hyperelliptic divisor (e.g. 
y is not hyperelliptic ), oJyiv + q) is essentially very ample by Lemma [3 . 9 1 and again we are done. 

It remains to consider the case where p + q\s cl hyperelliptic divisor on the 2-inseparable hyperelliptic 
curve Y, where we must determine ^. Then the linear system \coy{p + q)\is just SyvcSt] where r] = \p + q\ 
is the g\. On the other hand, \co{2p + 2q) \ is very ample and contains as a hyperplane Sym^^^t], which 
induces the same map to as rj (albeit as rational normal curve of degree g + 1). This hyperplane 
corresponds to a point zq on the secant line 'pq C P^^^ = P(H'^(a;y(2p + 2q)) which can be identified 
with the hyperelliptic azimuth. Because projection from 'pq itself corresponds to Sym'? rj (i.e. to the map to 
a rational normal curve of degree g), a projection from any z G 'pq, can fail to be an embedding off p + q 
is if the projection as a map on X coincides with the g2, i.e. coincides with projection from zq; but because 
\co{2p + 2q) I is very ample, this is only possible \t z = zq, i.e. the hyperelliptic azimuth. □ 

An azimuthally marked curve (Y, ^) is said to be hyperelliptic if all unimarks in ^ are co-hjrperelliptic 
and hyperelliptic on Y and all bimarks are co-hyper elliptic and hyperelliptic on Y and the associated 
azimuth in ^ is the hyperelliptic one. Then combining the above Lemma with the Residue Lemma ISJl we 
conclude 

Lemma 7.2. Let (Y, ^) be a 2-inseparable hyperelliptic azimuthally marked curve. Then the sepcanonical system 
a;|^'^P(Y, ^) is composite with the canonical system on Y. 

Proof. Part (i) of the Residue Lemma implies that we may assume ^ consists of a single (hyperelliptic) 
unimark or azimuthal bimark. Then the bimark case follows from the above Lemma, while the unimark 
case is straightforward. □ 

Our result main result on the sepcanonical system. Theorem [73] below, is that unless a stable curve is 
hyperelliptic, its sepcanonical system is 'essentially very ample' (see below); only 'essentially' because of 
some exceptional behaviour at the separators, but this turns out to be good enough for applications. The 
proof has mostly been given above, in the discussion of 2-inseparables. 

Definition 7.3. Let Y bea curve with some unimarks and bimarks, all smooth. A linear system (L, V) on Y is said 
to be essentially very ample if it induces an embedding on every length-2 subscheme, except 

(i) if p is a unimark, at least one of 

{L{-ip), V{-ip)), V{-ip) := y n H\L{-ip)), i = 0, 1,2,3 

induces an embedding on the subcheme 2p; 

(ii) if{p,q) is a bimark, at least one of{L{—ip — iq), V{—ip — iq)),i = 0, 1,2 induces an embedding onp + q. 

Another consequence of LemmaEHis 

19 



Lemma 7.4. Let 6 be a non-hyperelliptic azimuthal *-sep on X. Assume 6 is left-extremal, i.e. lX{9) contains no 
other *-sep of X. Then the sepcanonical system |a;x|^*^^ essentially very ample on the 2-component containing 

Proof. Note \,X{6) is inseparable and 2-inseparable relative to \^6. The cases where 9 is non-hjrperelliptic 
as plain *-sep are settled by Lemmas 13.61 and 13.101 The remaining case, where 6 is hyperelliptic but the 
azimuth is not, is settled by Lemma l7.1l □ 

Theorem 7.5. Let Xbea semistable middle-azimuthal curve of genus g >2. Then either 

(i) the sepcanonical system |<x'x|^^'' is essentially very ample on each 2-component ofX; or 

(ii) X is hyperelliptic and |a;x l^'^P maps each 2- component ofX 2:1 to a rational normal curve. 

Proof. First, it is easy to see that no generality is lost by assuming X is stable. By Proposition |3.11[ we may 
assume X is 2-separable. We will assume to begin with that X is of semicompact type. 

Assume X is (azimuthally) not hyperelliptic. Consider a 2-component Y of X that is an end-vertex of 
the separation tree G2(X). We may assume Y is left-extremal for X. Let Q be the edge into Y. We assume 9 
is a bisep as the sep case is easier. By definition, 9 is not (azimuthally) hyperelliptic. Therefore by Lemma 
17.41 |a;x|^'^J' is essentially very ample on Y. Now if 9 is the unique edge of G2(X), we are done by applying 
the same argument to its other vertex. Else, there exists another end-vertex Z with corresponding edge p, 
which we may assume is right-extremal. By the same argument, |a;x|^'^P is essentially very ample on Z. 
Because 9 c iX{p), iX{p) is not hyperelliptic. By induction on the number of components, \co j^x(p) 1^*^^ 
is essentially very ample , hence so is I'^xT'^x^o)' Therefore |c(JxI^'^^ is essentially very ample. 

It remains to consider the case where X is stable and not of semicompact type, hence contains a proper 
polyseparator 0. Such X is never hyperelliptic. The claim then is that loj'xl^'^'' is always essentially very 
ample. Suppose first that X contains a sep or maximal 2-sep, i.e. that the separation tree G2(X) is nontriv- 
ial. Let 9 be an edge. We may assume C RX(f?), which implies rX(0) is non-hyperelliptic . Replacing 
by a terminal edge to its left, we may assume Y = is an end-vertex of G2(X). Because rX(0) 

is non-hyperelliptic , \co ^x{e)T^'^ is essentially very ample by induction on the number of components, 
hence so is |a;x|^'^P| ^x{0)- Now if Y is non-hyperelliptic , \cox\^'^'^ is similarly essentially very ample on it. 
Otherwise, Y is hyperelliptic and in particular of semicompact type. Because, as an end-vertex, Y contains 
no sep or maximal bisep of X, it follows that Y is 2-inseparable relative to \^9. Then it follows from Lemma 
13.61 that |a;x|^'^P is essentially very ample on Y. 

Now we may assume G2(X) is trivial and X contains a proper polyseparator 0. It will suffice to prove 
that |a;x 1^*^^ is essentially very ample of each connected component Y of X®. We may assume 

y = lX(0),0 = (01,02) C = (0i,...,0„),n > 3. 

If Y is non-hyperelliptic, then |<jjy|^'^P is already essentially very ample, hence so is the bigger system 
|'^x|'*^^|y- Therefore assume Y is hyperelliptic. Note that 9_ is not right- hyperelliptic, so there is no left 
azimuthal condition attached to it. If 9_ is contained in a single 2-component of Y then Y is 2-inseparable, 
hence Ict^xl***^^ is essentially very ample on it by Lemma l3l2l Else, each 0,, f = 1, 2 is contained in a different 
2-component Y, of Y. Then because Y contains no other *-seps of X, it follows that Y must be 2-inseparable 
relative to l^i, l02/ so we can conclude by Lemma |331 □ 

Remark 7.6. In |13|, we show that the hyperelliptic middle-azimuthal stable curves are precisely the limits 
of smooth hyperelliptic curves. This is done by constructing a modification of the Hodge bundle on a 
suitable parameter space (a blowup of the Hilbert scheme), together with a map to the relative canonical 
bimdle, whose degeneracy locus meets the boimdary transversely. 
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